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Background of the Invention 

This invention relates generally to human/computer interfaces and 
more particularly to mechanical input devices for computerized systems. 

It has become increasingly common to computerize systems, from 
the trivial (e.g., the computerized toaster or coffee pot) to the exceedingly 
25 complex (e.g., complicated telecommunications and digital network 
systems). The advantage of computerization is that such systems become 
more flexible and powerful. However, the price that must be paid for this 
power and flexibility is, typically, an increase in the difficulty of the 
human/machine interface. 

3 0 The fundamental reason for this problem is that computers operate 

on principles based on the abstract concepts of mathematics and logic, 
while humans tend to think in a more spatial manner. People inhabit the 
real world, and therefore are more comfortable with physical, three- 
dimensional objects than they are with the abstractions of the computer 

3 5 world. Since people do not think like computers, metaphors are adopted to 
permit people to effectively communicate with computers. In general, 



PATENT 



INTIPOOIB 



-2- 



better metaphors permit more efficient and medium independent 
communications between people and computers. 

There are, of course, a number of human/computer interfaces which 
allow users, with varying degrees of comfort and ease, to interact with 
5 computers. For example, keyboards, computer mice, joysticks, etc. allow 
users to physically manipulate a three-dimensional object to create an input 
into a computer system. However, these human/computer interfaces are 
quite artificial in nature, and tend to require a substantial investment in 
training to be used efficiently. 

1 0 Progress has been made in improving the human/computer interface 

with the graphical user interface (GUI). With a GUI interface, icons are 
presented on a computer screen which represent physical objects. For 
example, a document file may look like a page of a document, a directory 
file might look like a file folder, and an icon of a trash can be used for 

15 disposing of documents and files. In other words, GUI interfaces use 
"metaphors" where a graphical icon represents a physical object familiar to 
users. This makes GUI interfaces easier to use for most users, GUI 
interfaces were pioneered at such places as Xerox PARC of Palo Alto, 
California and Apple Computer, Inc. of Cupertino, California. The GUI is 

20 also often commonly used with UNIX™ based systems, and is rapidly 
becoming a standard in the PC-DOS world with the Windows™ operating 
system provided by Microsoft Corporation of Redmond, Washington, 

While GUIs are a major advance in human/computer interfaces, they 
nonetheless present a user with a learning curve due to their still limited 

25 metaphor. In other words, an icon can only represent a physical object: it 
is not itself a physical object. Recognizing this problem, a number of 
researchers and companies have come up with alternative human/computer 
interfaces which operate on real- world metaphors. Some of these concepts 
are described in the July, 1993 special issue of Communications of the 

30 ACM, in an article entitled "Computer Augmented Environments, Back to 
the Real World." Such computer augmented environments include 
immersive environments, where rooms are filled with sensors to control 
the settings of the room, as researched at New York University (NYU) in 
New York, New York. Another example is the electronic white boards of 

3 5 Wacom and others where ordinary-looking erasers and markers are used to 
create an electronic "ink/' Wellner describes a "DigitalDesk" that uses 
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video cameras, paper, and a work station to move between the paper and 
the electronic worlds. Fitzmarice has a "Chameleon" unit which allows a 
user to walk up to a bookshelf and press a touch-sensitive LCD strip to 
hear more about a selected book. Finally, MIT Media Lab has a product 
5 known as Leggo/Logo which lets children program by snapping plastic 
building blocks together, where each of the building blocks includes an 
embedded microprocessor. 

Bishop has developed a "marble answering machine" which appears 
to store a voice mail message in a marble that drops into a cup. The 
1 0 marble, in fact, triggers a pointer on a small computer which stores the 
message. To play back the message, the marble is dropped into the 
machine again. This marble answering machine has been publicly known 
at least as of June, 1993. 

While strides have been made in attempting to improve 
1 5 human/computer interfaces, there is still progress to be made in this field. 
Ultimately, the interface itself should disappear from the conscious thought 
of users so that they can intuitively accomplish their goals without concern 
to the mechanics of the interface or the underlying operation of the 
computerized system. 
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Summary of the Invention 



The present invention improves the human-computer interface by 
using "interactors." An interface couples a detection field to a controller 
5 computer system which, in turn, may be coupled to other systems. When 
an interactor is entered into the detection field, moved about within the 
detection field, or removed from the detection field, an event is detected 
which, when communicated to the computer system, can be used to create a 
control signal for either the controller computer system or to a system 
1 0 connected to the controller computer system. Preferably, the detection 
field is suitably sized and configured so that multiple users can 
simultaneously access the field and such that multiple interactors can be 
engaged with the field simultaneously. 

By "interactor" it is meant that a physical, real world object is used 
1 5 that can convey information both to the controller computer system and to 
users. An interactor can provide identity (ID) information to the computer 
through an embedded computer chip, a bar code, etc. An object can also 
be made into an interactor by embedding higher-level logic, such as a 
program logic array, microprocessor, or even a full-blown 
20 microcomputer. An interactor forms part of a system wherein information 
is assigned by users to at least one object. 

An interactor system in accordance with the present invention 
includes a detection space and a number of interactors which can be 
manually manipulated within the detection space. The interactors 

25 preferably have a unique ID. An interface responsive to the interactors in 
the detection space provides signals to communicate information 
concerning the interactors (e.g. ID, position, EXIT/ENTER, and 
"temporal" information) to the computer system. The EXIT/ENTER will 
often be referred to as UP/DOWN when referring to a two dimensional 

30 detection field, since an interactor is entered by putting it down on the 
field, and is exited by picking it up from the field. Importantly, the 
computer system processes the information within a semantic context to 
accomplish a user-desired task. By "semantic", it is meant that the meaning 
of an interactor is dependent upon the context in which it is being used, 

3 5 both in terms of explicit and implicit assignments of function and content. 



PATENT 



INT 1 POO IB 



-5- 



As will be appreciated from the above discussion, a method for 
controlling a computerized system includes the steps of: a) providing a 
detection space; b) placing a physical, identifiable interactor having a 
semantic meaning within the detection space; c) determining the meaning of 
5 the interactor within the semantic context; and d) controlling a 
computerized system in response to the semantic meaning of the interactor. 

There are a number of specific applications^ for the interactor 
technology of the present invention. Two examples are given, one which 
allows for the control of an audio system to create a "virtual room", and 
10 the other which provides an event marking system for recorded media or 
other time based activities. 

In the first example, an audio system is provided which can bring a 
number of widely dispersed individuals together into a common auditory 
space. For example, the audio system can provide a "virtual room" in 

1 5 which individuals are brought together in the auditory sense from various 
locations. For example, individuals A, B, and C can be in separate physical 
offices, yet individual A might wish to casually chat with individuals B and 
C as if they were in the same office space. Individual A then uses 
interactors representing B and C (perhaps with their pictures on them) in a 

20 detection field to indicate that he wishes to converse with individuals B and 
C. The interactors detected by the detection field generate control signals 
within a controlling computer to control microphones, speakers, and 
amplifiers to make this happen. In this fashion, and by a very simple 
metaphor, A, B, and C can be made to inhabit the same "virtual room" for 

25 conversation and other auditory communication. 

In the second example, a videotape "marking" system is described. 
A videotape player is coupled to a controlling computer, and a videotape is 
played and observed by one or more users on a monitor. When an event 
occurring on the videotape is to be logged or marked, an interactor is 

3 0 engaged with the detection field. The controlling computer then retrieves 
timing information from the videotape player and combines this with the 
marking event. Removal of the interactor from the detection field can 
signify the end of the event, or can signify nothing, depending upon the 
context and the desires of the users. The detection field is preferably sized 

35 and configured so that multiple viewers of the video playback can 
simultaneously access the detection field. By taking a group approach, each 
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individual can be watching for and marking a specific event or a small 
group of events. This approach can reduce the fatigue and tedium with 
logging videotape. 

By using interactors, the human/computer interface is greatly 
5 enhanced. In the example of the audio control system, it takes little or no 
training to use the system since the interactors and their spatial 
relationships are intuitive to the user. Likewise, it is a very physically 
intuitive gesture for a user to place a labeled or otherwise evocative 
interactor on a detection field in response to a certain event detected in a 
1 0 video playback. The present invention therefore provides a more intuitive 
and richer metaphor for the interaction between humans and computerized 
systems. Furthermore, the present invention provides a system whereby 
multiple users simultaneously communicate with a computerized system 
using the metaphor. 

15 These and other advantages of the present invention will become 

apparent upon reading the following detailed descriptions and studying the 
various figures of the drawings. 
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Brief Description of the Drawings 



Fig. 1 is a pictorial representation of an interactor system in 
accordance with the present invention; 

5 Fig. 2 is a pictorial representation of a first preferred embodiment 

of the present invention; 

Fig. 3 is a side elevational view of a two-dimensional detection field 
in accordance of the present invention; 

Fig. 4 is a top plan view taken along line 4-4 of Fig. 3; 

1 0 Fig. 5 is a perspective view of an interactor in accordance with the 

present invention; 

Fig. 5a is a schematic representation of the internal circuitry of the 
interactor of Fig. 5; 

Fig. 6 is a schematic diagram of the circuitry of the detection field 
1 5 illustrated in Figs. 3 and 4; 

Fig. 7 is a flow diagram of a computer implemented process running 
on the microprocessor of Fig. 6; 

Fig. 8a is a data word produced by the process of Fig. 7; 

Fig. 8b is a table illustrating the meanings associated with the state 
20 bit of the data word of Fig. 8a; 

Fig. 9a illustrates a one-dimensional detection field; 

Fig. 9b illustrates both a three-dimensional and a four-dimensional 
detection field; 

Fig. 9c illustrates an alternative three-dimensional detection field; 

25 Fig, 10 illustrates an interactor used to control an audio system; 

Fig. 11a illustrates a first embodiment of the audio control system 
wherein the user is embodied into the system; 
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Fig. lib illustrates a second embodiment of the audio control system 
wherein the user is not embodied into the system, Le. is omniscient to the 
system; 

Fig. 11c illustrates a layout of a two-dimensional detection field used 
5 for the audio control device; 

Fig. 12 is a block diagram for an audio control system of the present 
invention. 

Fig. 13 is a block diagram representing the computer implemented 
processes running on the computers and server of Fig. 12; 

10 Fig. 14 is a flow diagram illustrating the operation of the application 

program of Fig. 13; 

Fig. 15 is a flow diagram illustrating the operation of the network 
library of Fig. 13; 

Fig. 16 is a flow diagram illustrating the server software operation 
15 of Fig. 13; 

Fig 17 is an illustration of inter actors on a detection field for 
marking events in temporal flows; 

Fig. 18 is a flow diagram of an event marker system in accordance 
with the present invention; 

20 Fig. 19 is a flow diagram illustrating the "Control Media Based On 

Event" step of Fig. 18; 

Fig. 20 is a flow diagram illustrating the "Process Binding Event" 
step of Fig. 18; 

Fig. 21 is a flow diagram of the "Mark Temporal Flow" step of Fig. 
25 18; and 

Fig. 22 is a flow diagram illustrating the "Process Other Event" step 
of Fig. 18. 
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Detailed Description of the Preferred Embodiments 



In Fig. 1, an interactor system 10 includes a detection space 12, a 
controller computer 14, and an optional system 16. A number of 
5 interactors 18 (which will be discussed more fully hereafter) may be 
engaged with, moved around in, and removed from the detection space 12. 
These interactors 18 in conjunction with the detection space 12 help define 
a human/computer interface that is intuitive, flexible, rich in meaning, and 
is well adapted for use by multiple simultaneous users. 

10 As used herein, the term "detection space" or the like will refer to 

any n-dimensional space in the physical world. The detection space will be 
alternatively referred to as a "detection field," an "event field/' and the 
like. Therefore, such terms as "space," "field," "domain," "volume," 
should be considered as synonymous as used herein. However, "field" will 

15 be used more frequently with respect to a two dimensional detection space, 
while "space" will be used more frequently with respect to a three 
dimensional detection space. 

Since we live in a three-dimensional world, any real- world detection 
space will have a three-dimensional aspect. However, if only two of those 

20 dimensions are used as input to the computer 14, we will refer to the 
detection field as a "two dimensional." Likewise, if only one-dimension is 
used as an input to computer 14, we will refer herein to such a field as 
"one dimensional." Furthermore, in certain embodiments of the present 
invention, the detection space may be time-variant, allowing the inclusion 

25 of four dimensional detection spaces. Various examples of detection spaces 
and fields will be discussed in greater detail subsequently. 

Computer 14 is preferably a general purpose microcomputer made 
by any one of a variety of commercial vendors. For example, computer 14 
can be a Macintosh computer system made by Apple Computer, Inc. or a 
30 PC/AT compatible DOS computer system made by Compaq, IBM, 
Packard-Bell, or others. Computer 14 is coupled to the detection space 12 
as indicated at 20 such that it may receive information concerning an 
interactor 18 placed within the detection space 12. An interface is 
provided between the detection space 12 and the computer 14 which may 
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be either internal to or external of the computer system 14. The design 
and implementation of interfaces is well known to those skilled in the art, 
although a preferred implementation of an interface of the present 
invention will be discussed in greater detail subsequently. 

By coupling the optional system 16 to computer 14, interactors and 
the optional system 16 can interact within controller computer 14. The 
system 16 may serve as an input to computer 14, an output from computer 
14, or both. When used as an input to computer 14, the system 16 can 
provide data on a line 22 which is used in conjunction with data on line 20 
derived from the interaction of an interactor 18 with the detection space 
12. When used as an output from the computer system 14, the system 16 
can be controlled by the interaction of the interactor 18 with the detection 
space 12. The system 16 can be of a standard commercial design (e.g. a 
videotape player), or can be a custom system designed for a particular use. 

An interactor system 24 used to mark events in a temporal flow is 
illustrated somewhat schematically in Fig. 2. The interactor system 24 
includes a detection field 26, a computer 28, and a video system 30. With 
the interactor system 24, a videotape or other video source can be 
displayed on a screen 32 of the video system 30 and events can be 
"marked" by engaging interactors 34 with the detection field 26. The 
images on video screen 32 may be recorded such as within a 
recording/playback unit 35 of the video system 30, or may be purely 
transitory images, such as those produced by a video camera 36 of the 
video system 30. If recorded, the images can be "marked" 
contemporaneously with recording of the image, or after the fact. In the 
latter instance, the unit 35 would simply be used in its playback mode to 
playback an earlier recorded video tape for event marking. 

The detection field 26 is, in this embodiment, a two-dimensional 
detection field in that it can detect positions of interactors 34 in both an "x" 
and a "y" direction. However, the detection field 26 of Fig. 2 does not 
detect vertical displacement from the detection field (i.e. in the z-direction) 
in this present embodiment. The detection field 26 is provided with four, 
V-shaped channels 38 which permit the interactors 34 to be engaged with 
the detection field 26 at a convenient angle. A number (e.g. 12) of 
interactors 34 can be engaged with each of the channels 38. 
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The detection field 26 is coupled to the computer 28 by an interface 
40. More particularly, a first cable 42 couples the detection field 26 to the 
interface 40, and a second cable 44 couples the interface 40 to the computer 
28. The construction and operation of both the detection field 26 and 
5 interface 40 will be described in greater detail subsequently. 

The video system 30 is coupled to computer 28 by a cable 46. 
Preferably, the computer 28 includes an internal video interface card 
which engages with a suitable connector at one end of the cable 46. Other 
embodiments have other arrangements for connecting the video system to 

10 the computer. Video systems 30 and video system interface cards (not 
shown) are commercially available from such sources as Radius 
Corporation of California. The video camera 36 can be coupled to the 
record/playback unit 35 by a cable 48, or can be directly coupled into the 
computer 28 through the aforementioned video interface card (not shown). 

1 5 Video cameras such as video camera 36 are available from a number of 
manufacturers including Sony Corporation of Japan. 

Fig. 3 is a side elevational view of detection field 26. Shown 
engaged with three of the four V-shaped channels 38 are interactors 34. 
Again, while only one interactor is shown engaged with each of channels 
20 38, a number of interactors 34 (e.g. 12) can be simultaneously engaged 
with each of the channels. The body 50 of the detection field 26 is 
preferably made from an insulating material such as wood or plastic. 

In a preferred embodiment of the present invention, a plurality of 
permanent magnets 52 are provided in a first wall 54 of each of the V- 

25 shaped channels 38 corresponding, one each, with positions where 
interactors can be engaged with the channels. The backs 56 of interactors 
34 are adapted to engage the walls 54 of the channels, i.e. preferably both 
the walls 54 of the channels and the backs 56 of the interactors are planar 
in configuration. Each of the interactors 34 are also provided with a 

30 magnet 58 which is attracted to a magnet 52 when the back 56 of the 
interactor 34 is engaged with a wall 54 of the V-shaped channel 38. This is 
accomplished by having opposing (N/S) poles of magnets 52 and 58 face 
each other when the interactor 34 is engaged with the channel 38. Since 
the magnets 52 and 58 are slightly offset in the vertical sense when the 

35 interactor 34 is engaged with the channel 38, a force F is exerted on each 
of the interactors 34 to firmly hold the back 56 against the wall 54 and to 
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firmly hold a base 60 of the interactor 34 against an abutting wall 62 of the 
V-shape channels 38. Therefore, the magnets not only hold the interactors 
34 in position, they also ensure good contact between abutting surfaces of 
the interactor 34 and channel 38. 

As seen in Fig. 4, each of the channels 38 are provided with a 
number of contacts 64 and a grounding strip 66. The contacts 64 are 
electrically conducting and are located in walls 54 of the channels. The 
grounding strips 66 are also electrically conducting and are connected near 
to the bottom of the walls 62 of the channels. As will be discussed in 
greater detail subsequently, an interactor 34 makes electrical contact with 
one of the contacts 64 and with the grounding strip 66 when properly 
engaged with the V-shaped channel 38. The magnets 52 and 58, in addition 
to urging the interactor 34 into the channel 38, also help assure that the 
interactor 34 is aligned properly in the x direction so that it makes good 
contact with the intended contact 64. This desired result is accomplished 
because the magnets 52 and 58 will create a force that will attempt to align 
the interactor in the x direction. The contact 64 and the grounding strip 66 
can be made, for example, from copper or any other suitable conductive 
material. 

In Fig. 5, a perspective view of an interactor 34 shows the base 60 
and back 56. The body 68 of the interactor 34 of Fig. 5 is a rectangular 
prism and is made from a non-conductive material such as wood or plastic. 
Base 60 includes a foil member 70 which is adapted to engage the 
grounding strip 66 of the V-shaped channels 38. Attached to the back 56 is 
a contact 72 which is adapted to engage one of the contacts 64 of the V- 
shaped channels 38. The foil 70 and contact 72 are made from a suitable 
conductive material, such as copper. 

The interactors 34 and the detection field 26 are sized for easy use 
and for the simultaneous use by several persons. For example, the 
interactors 34 can have dimensions of about 0.5 in. X 1.5 in. X 2.0 in., 
while the detection field can have dimensions of about 1 ft X 2 ft. X 3 in. 
in height. This permits the interactors 34 to be comfortably held in a 
user's hand, and allows multiple users to simultaneously interact with the 
detection field 26. 

In Fig. 5a, the internal circuitry of the interactor 34 is shown. The 
circuitry includes an identification (ID. chip 74), and a diode 76. The ID 
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chip is available from Dallas Semiconductor of Texas as part number 
DS2401, and provides a unique 48-bit identification (ID) when properly 
queried. The diode 76 prevents false keying, as is well known to those 
skilled in the art of keyboard design. The ID chip 74 is coupled to node 70 
5 by the diode 76, and is coupled to the contact 72 by a line 78. 

In Fig. 6, the internal circuitry 80 of the detection field 26 is 
illustrated. More particularly, the internal circuitry 80 includes the four 
grounding strips 66 and the contacts 64 described previously. The contacts 
64 are coupled together in rows by lines 82 and are coupled to Vcc (e.g. 5 

1 0 volts) by pull-up resistors 84. Nodes of the circuitry 80 between the pull- 
up resistors 84 and the contacts 64 form a 12 bit bus which are input into a 
buffer register 86. Likewise, grounding strips 66 are coupled into a four- 
bit bus and are input into the register 86. A microprocessor 88 (such as an 
M68H141 made by Motorola of Austin, Texas) communicates with the 

15 register 86 via a bus 90. Collectively, the register 86 and the 
microprocessor 88 comprise the interface 40, and the 12 bit bus and the 4 
bit bus collectively forms the bus 42. The output bus 44 is under the 
control of microprocessor 88. It will be appreciated by those skilled in the 
art that the interface 40 will also include other well-known components, 

20 such as RAM for scratch-pad storage, ROM to store the control 
instructions for the microprocessor 88, etc. 

In Fig. 7, a computer implemented process 92 that runs on the 
microprocessor 88 to control the circuitry 80 will be described. The 
instructions for this process 92 are stored in the aforementioned ROM of 

25 the interface 40, as will be appreciated by those skilled in the art. The 
process 92 begins at 94 and, in a first step 96, the microprocessor 88 clears 
the bit maps. By bit map, it is meant binary digits are mapped to particular 
locations on the board. These bit maps are preferably stored in the 
aforementioned RAM memory of the interface 40. Next, in a step 98, the 

30 rows and columns of the circuitry 80 are read and put into the current bit 
map. Next, in a step 100, a debounce routine is performed. Debounce 
routines are well known to those skilled in the art of keyboard and 
computer switch design. Next, in a step 102, locations on the detection 
field 26 that have "pieces" are "marked" on the bit map. As used herein, a 

35 "piece" is an interactor. By "marking" a location, a piece is engaged with 
the detection field 26 such that it makes electrical contact with the circuitry 
80. Next, in a step 104, the ID of each of the pieces engaged with the 
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detection field 26 is read, and is then compared to a list of the pieces (also 
stored in RAM memory of the interface 40). In a step 106, if a newly read 
ID is not equal to an old ID for a particular position, then a report is made 
that a new piece has been put down (i.e. engaged with the detection field 
5 26) and an old piece has been picked up from the same position. This 
information is added to a queue stored in RAM. In a step 108, if the same 
piece is in the same position on the detection field, it is reported that the 
piece is still down. In a step 1 10, if a new piece is detected at a position, it 
is reported that a piece has been placed on to the detection field 26. Next, 

10 the bit map is scanned in a step 112 for removed pieces and, if a removed 
piece is detected, the ID is reported. Next, in a step 114, the current bit 
map is moved into the older bit map. Subsequently, a step 116 determines 
if there is a user request. If not, process control is returned to step 98. If 
there is a user request, that user request is handled in a step 118. In the 

1 5 current preferred embodiment, this involves processing the user request to 
handle the commands "get board state", "get next event", "get all events", 
"erase all events", and "get board type (version)." After the user request 
has been processed, process control is again returned to step 98. 

In Fig. 8a, a digital word 120 of the present invention includes a 
20 number of bits. More particularly, the current word includes 55 bits. Of 
the bits, a bit B0 indicates the state, bits B1-B2 indicate the row, and bits 
B3-B6 indicates the column of the interactor. Finally, bits B7-54 hold the 
48 bit ID of the interactor. This data can be passed to the computer 28 via 
bus 44. 

25 In Fig. 8b, a table of state changes is shown along with their 

associated meanings. As described previously, the word 120 includes a 
state which is essentially exit/enter (up/down) for a particular interactor 
(i.e. when and how long has an interactor been positioned in the detection 
field). If the current state value is equal to 0, and the last state value is 

3 0 equal to 0, the meaning is that there is no piece (interactor) at that row and 
column position. If the current state is 1 and the last state is 0, that means 
that a piece has been put down at that row and column position. If the 
current state is 1 and the last state is 1, that means that the piece is still 
down since that last time that the detection field was scanned. Finally, if 

35 the current state is 0 and the last state is 1, that means that a piece has been 
picked up, i.e. an interactor has been removed from the detection field. 
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Figs. 9a, 9b, and 9c illustrate three alternative embodiments for a 
detection field. In Fig. 9a, a detection field 122 allows an interactor 124 to 
be linearly placed in a multiplicity of positions along an x axis. This is an 
illustration of a one-dimensional detection field. It should be noted that, at 
5 the trivial extreme, if the detection field 122 is shortened sufficiently, it 
can be made just large enough to accept a single interactor 124. This 
would comprise a zero-dimensional detection field which would simply 
detect the presence or absence of an interactor and its ID number, i.e. it 
can operate as a simple switch. 

10 In Fig. 9b, a detection space 126 is illustrated that can accept 

interactors 128 in three dimensions, i.e. along x, y, and z axes. The x, y, 
and z positions of an interactor 128 can all be used to determine the context 
or meaning of the interactor. For example, the base platform 130 can have 
a different meaning from a first platform 132, a second platform 134, a 

15 third platform 136, a fourth platform 138, and a fifth platform 140. 
Platform 136 could, for example, be dedicated to changing the identity of 
one of the interactors 128. Objects on platform 138 could be "enclosing" 
interactors on platform 140. The meaning and relationships of the various 
platforms can therefore be designed based upon desired functionalities 

20 specified by a user. 

It should also be noted that a fourth dimension can be added to the 
detection space 126 of Fig. 9b. In other words, the detection field 126 can 
change with time. One way to accomplish this is to allow the platforms to 
move over time such that their meanings change. For example, as 

25 indicated by arrows 142, 144, and 146, platforms 134, 136, and 138 can be 
allowed to move up and down, respectively, perhaps under the control of a 
motor (not shown). This permits an interactor 128 to have different 
meanings over a period of time. For example, the interactor 128 on 
platform 134 could represent a volume level for a loudspeaker which will 

30 diminish over time as the platform moves downwardly in a z direction. 
Therefore, it should be clear from the foregoing that the detection spaces 
or fields can be n-dimensional where n is 0, 1, 2, 3, etc. 

In the previous examples of detection fields and spaces, the detection 
fields and spaces have always been mapped by Cartesian (x, y, z) 
35 coordinates. In Fig. 9c, a detection space 148 in the form of a spherical 
globe is provided where a number of interactors have been adhered (such 
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as by magnets) to its surface. With such spherical detection spaces or 
fields, it may be more convenient to determine the position of the 
interactors using a spherical coordinate system. It should also be noted that 
other forms of detection fields can be provided including detection fields of 
5 irregular shapes. 

The present invention will be described more particularly in the 
form of the following two examples. It will be appreciated, however, that 
there are many other applications in which the interactor methods and 
systems can be used with good effect. 

10 Example 1: An Audio Control System 

In Fig. 10, an interactor 152 is shown which will be used for a 
particular implementation of the present invention. The interactor 152 
includes a body 154 that operates functionally in a fashion very similar to 
that of the interactor 34 illustrated in Figs 5 and 5 a. The interactor 152 
1 5 can be used with a detection field similar to or identical with detection field 
26 as illustrated in Figs. 3, 4, and 6. The detection field 26 used with the 
interactor 152 can also use the same interface 40 to interconnect the field 
with a computer system 28. 

The difference between interactor 152 and the previously described 

20 interactor 34 is therefore design related and not computational in nature in 
that they support different metaphors. With the interactor 152, a doll's 
head 156 or other talisman is provided with a peg 158 which can engage a 
hole 160 in the body 154. A small piece of white board 162 is removably 
attached to the body 154 by a pair of hook-and-pile (e.g. Velcro®) 

25 members 164a and 164b. The hook-and-pile member 164a is attached to a 
surface of body 154 while member 164b is attached to the back of the 
white board 162. In this way, the white board 162 can be removably 
attached to the body 154 of the interactor 152. A name, label, or other 
indicia can be provided on the white board 162 with a marker 166 as 

30 illustrated by the name "Fred." Therefore, the interactor 152 can be used 
to represent a person named Fred both by means of the head configuration 
156 and the name on the white board 162. It is a useful feature of the 
present invention in that interactors can be given distinct visual, aural or 
other sensory identities which aid in the metaphor of the human-computer 

35 interface. 
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In Fig. 11a, a detection field 166 has a number of interactors 168 
that can be positioned at various locations. In this instance, one of the 
interactors 168a represents the user herself. The other interactors 168 in 
this example represent other people. As noted, the pieces can be moved 
5 around such that their relative x, y positions change with respect to each 
other. It is therefore possible with the interactors of the present invention 
to create a "virtual room" wherein the utterances made by various persons 
represented by the interactors appear to be spatially located as indicated by 
the interactors. Therefore, the interactors and detection fields of the 
10 present invention can be used as a controller for forming groups in a 
"virtual room" and for varying the relative location of the various 
members of the group. 

For example, in Fig. 11a, before the user's interactor has been 
moved, two people would appear to be talking to the left of the user and 

1 5 two people would appear to be talking in front of and to the right of the 
user. After the interactor 168a has been moved to the new position 168a\ 
the two people that were apparently to the left of the user would now be 
behind the user, and the two people that were to the front and right of the 
user would be directly to the right of the user. By removing any one of 

20 the interactors 168 from the "virtual room," that person would no longer 
be part of the conversation, and removing the user's interactor 168a from 
the room (i.e. removing the interactor from the detection field 166) would 
eliminate the "virtual room." Of course, a suitable number of loudspeakers 
would be required to create the desired illusion. 

25 In Fig. lib, a slightly altered detection field 166' is used for 

substantially the same purpose as previously described. However, in the 
previous embodiment, an interactor representing the user herself is within 
the detection field 166, but in the embodiment of Fig. lib, the user does 
not have an interactor representing herself on the detection field 166'. In 

30 the previous embodiment as illustrated in Fig. 11a, the user is said to be 
"embodied" in that she is on the detection field and can move on the 
detection field relative to other interactors. However, in the "non- 
embodied" or "omniscient" version shown in Fig. lib, the position of the 
user is fixed at some point, either off or on the detection field 166'. For 

35 example, the user might be positioned at a point 170 just off of the 
detection field 166'. However, the other people represented by interactors 
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168 can be adjusted relative to the user to obtain much of the effect 
obtainable by the embodiment illustrated in Fig. 11a. 

In Fig. 11c, a potential "layout" of a detection field 166 is illustrated. 
If an interactor is placed near the back of the field, the volume associated 
with the person represented by that interactor is at its softest. Placing the 
interactor near the front of the field will make the associated person the 
loudest. Special positions on the left and right edges and down the center 
of the detection field can perform special functions, such as "pan", "get 
info", or "assign." 

In Fig. 12, an interactor system 172 in accordance with the present 
invention includes an audio server 174 and a number of workstations 176. 
As a system, the interactor system 172 can perform the functionality 
described with respects to Figs. 11a and lib. 

The audio server 174 includes a data server 178, a MIDI timepiece 
180, a number of MIDI devices 182, and an audio concentrator 184. The 
data server 178 receives data from a network bus 186 and is connected to 
the MIDI timepiece 180 by a bus 188. The MIDI timepiece 180 is 
connected to a rack of MIDI devices 182 by a bus 190, and the output of 
the MIDI devices 182 are coupled to the concentrator 184 by a bus 192. 
The concentrator 184 has, as inputs, a number of audio lines 194. 

Each workstation 176 includes a computer 196, interfaces 198, and 
detection fields 200 as described previously. The detection fields 200 can 
have one or more interactors 202 placed upon their surfaces as previously 
illustrated and described with reference to Figs. 11a and lib. The 
workstation further includes a pair of stereo loudspeakers 204 and a pair of 
stereo microphones 206. The loudspeakers 204 are coupled directly into a 
control box 208 which include loudspeaker amplifiers. The microphones 
are coupled to a pre-amplifier 210 which, in turn, are coupled to the 
control box 208. The control box 208 also includes microphone 
amplifiers. The audio lines 194 carry the microphone signals to the 
concentrator 184, and the loudspeaker signals from the concentrator 184 to 
the various speakers 204. 

The software operating the interactor system 172 is conceptually 
illustrated in block diagram form in Fig. 13. The databus 186 carries the 
data necessary to interconnect the various components of the system 172 
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and can, for example, be implemented on an Apple LocalTalk or Ethernet 
network protocol. It should be understood, however, that other network 
protocols such as Novell Netware or custom network software can also be 
used to provide the networking functions of the network bus 186. 

Three software routines are used to implement the interactor system 
172 of the present invention. Namely, each of the workstations 176 
operate an application program and a network library, and a data server 
178 operates data server software and the network library. The application 
program 212 runs on the computer 196 of each of the workstations 176 
that are part of the interactor system 172. Network libraries 214 likewise 
each run on a computer system 196. The network library communicates 
with the network bus 186 via a conceptual link 216 and with the application 
program via a conceptual link 218. The application program 212 
communicates with the network bus 186 via a conceptual link 220. The 
links 216, 218, and 220 are considered conceptual in that they are not 
physical links to the bus but, rather, logical links through operating system 
software, network software, internal buses, network cards, etc. 

The software running on the data server 178 includes the network 
library 222 and the data server software 224. The network library has a 
conceptual link 226 to the network bus and a conceptual link 228 to the 
data server software 224. The data server software has a conceptual link 
230 to the network bus 186. 

In the present implementation, the conceptual links 220 and 230 
from the network bus 186 to the application programs 212 and to data 
server software 224, respectively are AppleEvents created by an Apple 
networking system. The conceptual links 216 and 226 between the network 
library and the network bus 186 are preferably standard AppleTalk or 
Ethernet data packages. 

Fig. 14 illustrates the application program 212 running on the 
workstations 176. The process 212 begins at 232 and, in a step 234, the 
process is initialized with the board type, and the current board space. 
Next, in a step 236, the state of all persons on the system is communicated 
based upon the board state and board type. After the initialization and 
communication steps, the process enters an event queue 238 to await the 
next event. If the next event is a "pick-up" event, a step 240 determines 
whether the interactor is in a control space. As used herein, a control 
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space is a dedicated portion of a detection field used to control the process. 
If it is an assignment control space (where meanings are assigned to 
interactors) control is returned to the step 238. If the control space is a 
people control space, the audio is cut off in a step 242 and process control 
5 is again returned to step 238. 

If step 240 determines that the interactor is not in a control space, it 
is determined in step 246 if the board (i.e. the detection field) is self- 
embodied. If yes, it is determined in a step 248 if the interactor 
representing the user ("self) has been removed from the board. If not, the 
10 system provides an audio feedback and a new state to the server in a step 
250. If the interactor representing the user has been removed from a self- 
embodied board, a step 252 provides audio feedback and turns off the 
sound to all users. 

If the event queue detects an interactor being put down on the 
15 detection field, a step 254 determines whether it was put down into a 
control space. If yes, people information is provided in a step 256. If it 
was put into an assignment space, a step 258 inputs the assignment to the 
interactor. After either step 256 or 258 are completed, process control is 
returned to step 238. Next, in a step 260, it is determined whether there is 
20 a self-embodied board. If yes, a step 268 determines whether an interactor 
representing the user has been placed on the detection field. If not, or if 
step 260 determines that is not a self-embodied board, a step 264 provides 
audio feedback and resets the data concerning the person represented by the 
interactor. Otherwise, step 268 determines an interactor representing the 
25 user has been placed on the detection^field, audio feedback is provided, and 
a reset of all of the people represented by the interactors on the board is 
initiated. After steps 264 or 266 is completed, process control is returned 
to step 238. 

In Fig. 15, the functionality of the network library 214 of Fig. 13 is 
30 shown in greater detail. The functionality of network library 222 is 
substantially the same. The process 214 begins at 268 and, in a step 270, it 
is^ determined whether a function call has been received. If not, the process 
214 goes into an idle loop awaiting a function call. If a function call 
"receive event" has been received, a step 272 provides a requesting 
35 program with the information regarding the event. If a functional call 
corresponding to "send event" is received, an AppleEvent is created from 



PATENT 



INTIPOOIB 



-21 - 



the function call to communicate with other programs in a step 274. 
Process control is returned to the function call event loop 270 after the 
completion of either steps 272 or 274. 

In Fig. 16, the operation of data software server 224 of Fig. 13 is 
5 illustrated in greater detail. A process 224 begins at 276 and, in a step 278, 
it is determined whether an AppleEvent has been received. Again, this 
process 224 is Macintosh® specific, and like or equivalent processes can be 
used in other types of computer systems. If an AppleEvent has not been 
received, the AppleEvent loop 278 repeats until that AppleEvent is 

10 received. If the AppleEvent is a "value change" AppleEvent, a step 280 
determines whether there is a privacy violation. If yes, a step 282 notifies 
an error handler to handle the privacy violation. Process control is then 
returned to step 278. If there is not privacy violation detected by step 280, 
there is an update of dynamic information database in a step 284. Next, in 

15 a step 286, MIDI data is calculated and sent. In a subsequent step 288, 
users with a vested interest in the function are notified, and process control 
is returned to step 278. If a "value inquiry" AppleEvent is detected, a step 
290 determines whether there is a privacy violation. If yes, a step 292 
notifies the network library with the error function and process control is 

20 returned to step 278. If there is not a privacy violation as determined by 
step 290, information is retrieved from the database in a step 294. Finally, 
in a step 296, the network library is called to reply and process control is 
returned to step 278. 

Example 2: Videotape Marking System 

25 In this second example, an interactor system such as interactor 

system 24 is controlled to "mark" or "log" events in a videotape. In Fig. 
17, a detection field 298 includes three zones 300a, 300b, 300c and a 
number of interactors 302. Each of the interactors has a semantic meaning 
due to its identity, due to their position in the various zones 300a, 300b, 

30 and 300c of the detection field 298, and due to their amount or "type" of 
time they have been present in the detection field (up/down or, as 
sometimes referred to herein, exit/enter). The various objects 302 can be 
used mark and control the temporal flow of a recorded medium as 
described previously with regards to Fig. 2. 

35 As used herein, "temporal flow" will refer to the flow of events, 

either in real time or in some other time related context. Therefore, either 
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events can be marked in a temporal flow, or events that have been 
previously recorded or that are being concurrently recorded can be 
marked in the temporal flow. The "marking" may only be literally 
temporal (such as in real time), temporal with regard to a specific origin 
5 (such as seconds since the start of the tape), or temporal only in the sense 
that the measure could be translated into a temporal stream (such as feet of 
tape or frame number). While the present example relates to a recorded 
video medium, the marking and control of the temporal flow of another 
medium, such as an audio medium, may also be carried out. 

10 In Fig. 18, a computer implemented process 304 operating on a 

computer 28 of Fig. 2 for marking and controlling a temporal flow begins 
at 306 and, in a step 308, it is determined whether a non-board event has 
been received. If so, this "other" type of event is processed in a step 310 
and process control is returned to step 308. Next, in a step 312, it is 

15 determined whether a board event has been received. If not, process 
control is returned to step 308. If a board event has been received, the 
board is polled in a step 314 and it is determined in a step 316 whether a 
null board event has been received. A null board event may be that no 
interactors have been perceived in the detection field, or that no changes 

20 have been detected in the state of the interactors in the detection field. If 
so, process control returns to 308. However, if a board event has been 
received (i.e. it is not a null board event), the board event is parsed in a 
step 318. Next, in a step 320, it is determined the event type based upon 
any combination (e.g., any one, any two, or all three) of the interactor' s 

25 ID, location, and whether it is up or down (i.e. the time period of the 
interactor in the detection field). Next, in a step 332, the parsed event is 
processed by type. If it is a media event, a step 324 controls the media 
based upon the event. If it is a binding event, a step 326 processes the 
binding event. If it is a marking event, a step 324 marks the temporal 

30 flow. In this instance, the temporal flow is marked by receiving frame 
information from the video player and storing that frame information 
along with the event type in a database on the computer 28. If the event 
type is unrecognized, or after steps 324, 326, or 328 have been processed, 
process control returns to step 308. 

35 Step 324 of Fig. 18 is illustrated in greater detail in Fig. 19. To 

control a media based upon the event, process 324 begins at 320 and, in a 
step 332, the meaning of the interactor in the detection field is determined 
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from its ID, its location, and whether it is up or down. The meaning may 
be determined based upon any combination of the presence of the 
interactor (up/down), the ID of the interactor, and the location of the 
interactor. Next, in a step 334, this meaning is converted into control 
5 commands (e.g. stop, fast-forward, speed etc.) for the media system. The 
process 324 is completed at 336. 

Step 326 of Fig. 18 is illustrated in greater detail in Fig. 20. Process 
326 begins at 338 and, in a step 340, the current meaning of the interactor 
is displayed. The system then determines whether the user wants to 

1 0 redefine the current meaning of the interactor in a step 342. If not, the 
process 326 is completed as indicated at 352. If the user does wish to 
redefine the meaning of a particular interactor, it is determined in a step 
344 what type of redefinition is desired. If the meaning of the location is 
"re-bind the ID", then a step 346 redefines the binding of the ID to the 

15 meaning. If the meaning of the object is "re-define the location' 1 , the 
system redefines the binding of the location to the meaning in a step 348. 
If the meaning of the location or the ID is "re-define the proximity", a step 
350 is redefines the binding of the proximity to the meaning. As used 
herein, a definition for "proximity" is a measure of distance between the 

20 interactor and the detection field, or the position of an interactor in a 
detection space. After the completion of the binding steps of 346, 348, or 
350, the process 326 itself is completed as indicated at 352. 

The step 238 of Fig. 18 is illustrated in greater detail in Fig. 21. 
The process 328 begins at 352 and, in a step 354, the current temporal 
25 value of the media is retrieved. Next, in a step 356, a mark is stored with a 
temporal value and meaning based upon the ID, location, and how the 
interactor has been placed "up or down" in the detection field. The process 
328 is then completed at 358. 

The step 310 of Fig. 18 is illustrated in greater detail in Fig. 22. 

30 The process 310 begins at 360 and, in a step 362, an event type is 
determined. If the event type is board control, a step 364 issues board 
control commands. If the event type is "database," a step 366 manipulates 
the database of marks. If the event type is "navigate media," a media 
control command is issued by step 368. If the event type is "device 

35 control," a device to control is selected in a step 370. After the completion 
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of steps 364, 366, 368, or 370, the process 310 is completed as indicated at 
372. 

While this invention has been described in terms of several preferred 
embodiments and two specific examples, there are alterations, 
permutations, and equivalents which fall within the scope of this invention. 
It is therefore intended that the following appended claims be interpreted as 
including all such alterations, permutations, and equivalents as fall within 
the true spirit and scope of the present invention. 
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